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Abstract —The convergence of the generalized alternating pro¬ 
jection (GAP) algorithm is studied in this paper to solve the 
compressive sensing problem y — Ax + e. By assuming that 
AA^ is invertible, vre prove that GAP converges linearly within 
a certain range of step-size when the sensing matrix A satisfies 
restricted isometry property (RIP) condition of 82 k, where K 
is the sparsity of x. The theoretical analysis is extended to the 
adaptively iterative thresholding (AIT) algorithms, for which the 
convergence rate is also derived based on 52k of the sensing 
matrix. We further prove that, under the same conditions, the 
convergence rate of GAP is faster than that of AIT. Extensive 
simulation results confirm the theoretical assertions. 

Index Terms —Compressive sensing, generalized alternating 
projection, convergence, restricted isometry property, iterative 
shrinkage/thresholding algorithms. 

I. Introduction 

Consider the compressive sensing 0-0 problem 

y = Ax + e, (1) 

where A G '^mxn sensing matrix and usually M N-, 
X G is a sparse signal with K nonzero entries (iC-sparse), 
and e G denotes the additive noise. Compressive sensing 
aims to find the sparsest solution of x. To solve this problem, 
extensive algorithms have been proposed. Due to the 

foundational work of 0, 0, various algorithms 0-113 have 
been developed to solve the relaxed problem 

min||a;||i, subject to y = Ax + e, (2) 

X 

where ||a;||i = \^n\ L the .fi-norm, i.e., the summation 

of absolute values of each entry in x. 

The solution of 0 has been shown to be the sparest 
solution d, d, if the sensing matrix A satisfies the 
restricted isometry property (RIP) condition; 

(l-5s)||^||Bl|Asat|li<(l + <5s)||a;||i (3) 

where Ag is a subset of A constituted of S columns from A, 
and 5s G (0,1) is the RIP constant. Different conditions 0, 
d’ d^ d have been studied on the value of 5s for the 
guaranteed recovery of the sparse signal x. 

In this paper, we solve 0 via the generalized alternating 
projection (GAP) algorithm d- Specifically, we introduce 
the step-size parameter into GAP and propose the adaptively 
GAP algorithm. We prove in Section that GAP converges 
linearly within a certain range of the step-size, rather than 
the fixed step-size as proved in d- Connection of GAP 
and the adaptively iterative thresholding (AIT) algorithms 0, 


pT| , p8| , p9| is presented in Section [TVl and the theoretical 
analysis is also extended to AIT. We compare the convergence 
rates of GAP and AIT under the same RIP condition of the 
sensing matrix in Section |V] Extensive simulation results are 
provided in Section]^ to verify the theoretical assertions. 

II. The Generalized Alternating Projection 
Algorithm 

The generalized alternating projection (GAP) algorithm, 
originally proposed in p7) , has achieved excellent results in 
diverse compressive sensing systems in real world applica¬ 
tions ||20|-||3^. In the following, we first review the original 
GAP algorithm and then introduce the parameter of step-size 
into GAP. 

A. Review the GAP algorithm in d 

The original GAP proposed in d is developed to solve 
the weighted group £ 2,1 problem. Here we simplify it to solve 
the £1 problem in 0. 

GAP solves the equivalent problem of 0: 

min subject to ||a;||i < i? and Ax = y. (4) 

x,R 

where R is the radius of the £i-ball. 

GAP solves 0 as a series of alternating projection problem: 

{wt, 9 t) = argmin^lltu - 0II2, 

w,6 Z 

subject to ll^lli < and Aw = y, (5) 

which is equivalent to 

{wt,9t) = argmini||uj-0||2-f Aill^lii, 

Z 

subject to Aw = y, (6) 

where At is the regularized parameter at t-th iteration with t 
denoting the iteration of the algorithm; it is related to jn 
0^ please referring to d for details. 

By assuming that AA^ is invertible, 0 is solved by 
alternating updating w and 9 in flT) as below 

lut+i = 9t + A^{AA^)-\y-A9t), (7) 

9t = tut©max(l-(8) 

I \wt\ j 

where © is the element-wise product operator and At is the 
shrinkage threshold at f-th iteration. 
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B. Adaptively GAP via Parameterizing the Step-size 

Rather than the GAP developed in pT) , which is based 
on the fixed step-size {a = 1), we introduce the step-size 
parameter a to and then an adaptively GAP solver now 
becomes: 

wt+i = Ot + aA^- AOt), (9) 

Ot = utt © maxi 1 - y^,o| . (10) 

I \wt\ J 

In this work, we consider A( selected as follows: 

Ai = -VI J (11) 

Wt = sort (I lUt I, ‘descend’), (12) 

where denotes the (m* + l)-th entry of Wt, which 

sorts the absolute value of Wt from large to small. We need 


m* > K. 


(13) 


0 < 62 K < 1) 


(14) 


then X* is the unique sparsest solution. In our work, GAP 
provides a series of with m*-sparse, and we assume 


0 < 5 




< 1. 


(15) 


Since m* > K, requirement ( [T5] l implies that 0 < 52k < 1 is 
always satisfied in our case. Without confusion, we use both 
5 and SK-vm* in the following derivation (S = Sx-i-m*)- We 
prove the convergence of GAP based on Sk-vtu* in Section III 
and this proof is extended to AIT in Section]^ Comparison 
of the convergence between GAP and AIT is presented in 
Section 


III. Convergence oe the Adaptively GAP 
Let us start the derivation from (|^ 

Wt+i = et-\-aA^{AA^y^{y-Aet), (16) 
and recall that x* is the true AT-sparse solution 

y = Aa:*-fe. (17) 

We have 

Wt-vi - X* = Ot - X* Q;A^(AA^)“^(y - A0t) 

= et-x* + aA^{AA^)-\Ax* + e- A6>t) 
= et-x* - aA^iAA^y^AiOt - x^ 

+ aA^{AA^)-y. 


A. Noiseless Case 

Firstly consider the noiseless case, i.e., e = 0. 0 becomes 
wt+i - X* = Gt - X* - aA^{AA^)-^A{et - x*) (19) 
Taking the ^ 2 -norm on both sides, we have 

\\w,+,-x*\\l = \\9,-x*\\l 

+ (a^ - 2a)||A^(AA^)-iA(0t - x*)\\l. (20) 

Now taking account of AA^ being invertible. 


AA^ UAU^, 


( 21 ) 


A = diag{ei,..., cm} and UU^ = Im,(22) 

where 1m is the M x M identity matrix. 

We further define 

ma.v clef 


= max{ei,...,eM}, 

I yy s/ey j 


A 2 = diagl 


Similar selection of m* can also be found in the literature for 
AIT algorithms; for example, m* = K is used in 0- 
According to fTS) , let x* be a A-sparse solution of the 
equation y = Ax, if the sensing matrix A satisfies the RIP 


Equation (20i becomes: 


(23) 

(24) 

Wwt+i-x'^Wl = \\et-x'‘\\l-\-{y-2a)\\A~iU^A{et-x*)\\l. 

(25) 

Introduce the following Lemma: 

Lemma 1 ^ A satisfies R/P|^UA also satisfies RIP if IJ 
is an orthonormal matrix (ULL =1). 

Proof: 

||UAsa;||2 = at^AjU^UAsa; = ai^AjAsa: = IjAsatsU^. 
Since 

(l-5)|lai||2<||Asat||l<(l-f5)||ai||i 

we have 

(l-5)||ai||B||UA5ir|!B(l + <5)|!at||r 


Recall (|25ll and using RIP 


A-5U' A(0t - X* 


l2> 


\ij^A{et-x*)\\l 


■>. II2 

— ^ma.v II ^ ll'^’ 


(26) 

(27) 


where the RIP is based on the non-zero entries of {Gt — x*). 
Since we have imposed in ([T0|-( 11 1 that Gt has at most m* 
nonzero entries, {Gt — x*) has at most (to* -f K) nonzero 
elements. Thereby 5 = 5m- +k as mentioned in Section [11] 
Considering a G (0, 2), we have — 2a < 0, and from 
@ and 

{y-2a){l-5) 


Imt+i - a;*||^ < 


1 + 


Gt-x*\\l. (28) 


Note that when A is fixed, is fixed and 5 is also fixed 
given K and to*. If we can find the relationship between ||0t — 
a;* 11 2 and ||mt — a;*|| 2 , we have the convergence condition of 
GAP. In order to do so, we first introduce the following lemma: 


( 18 ) 
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Lemma 2 For any x,y G K^, 

ll=^ + y|li<2(||a:||^ + ||y|i^). 

Proof can be found in | fT8| and thus omitted here. 
We further define the following sets: 

Definition 1 

1+ : Vi, that |a;*| > 0, 
where x* denotes the i-th entry of x*. 

Definition 2 

: Vi, that \6tf > 0, 
where 9t^i denotes the i-th entry of Ot- 
With these definitions: 


(ii) 1+ ^ since we only consider entries in 

there exists io S but ip 

X* 

-\-l — ^t,iQ ^io\ 

< max Iwti — x*\. 


it) 


\0t-x*\\l 


(29) 


(30) 


(31) 


= ||0,,j(o-at^<oll2 + ll®;^\^(oll2(32) 


i^Ji 


(t) 


(40) 


Following this, based on (37 1 , 

j.u)||2 < 2TO*||mt - a:*||2. (41) 

3) The third term: 

Aw < 2\I+\J^\ max |u;t,,|^42) 


< 2|Z+\J-f|A2, (43) 

where \I+\J^'^\ denotes the number of elements in the 
set and because m* > K, we have 


|i+uf I < \Jr\^+\. 


(t)\ 


(44) 


where 6 
(similarly for x* (jj) and x 


^ j{t) denotes all the entries of Ot in the set 


(t) 

+ 




lAJr 


(t) 


denotes the entries of x* 


in 1+ but not in . From (32i and using Lemma we have 

\\^t-x*\\l = -a:jwll 2 

~ 4"II 2 (33) 

— ~ II 2 + ~ II 2 

4" ~ ^x+\jf'> II 2 4" 2||t<t(_2:+\j-|*> II 2 


= 2|lm 


t,jf'>ux+ 


— X 






+ 2\\e 

4" Hz- 


(34) 


Separately consider the three terms on the right-hand side of 
1) The first term: 


2||m 

2) The second term: 




I 2 < 2 ||K;t - x*"^ 


2- 




2 = 2m* A? 


(35) 


(36) 


<2m*\\w^ ^w-<(oll 2 , (37) 


where Xj < ||u;^ - at* (t)||i is from 


At 


t < max - a;*|. 


IGJ. 


(t) 


(38) 


This can be proved by considering the following two 
cases: 

(i)Z+c4‘): 

At — +1 — |tCt,rn*-t-l| — |tht^7n*-|-l 

< max jrct.i — a:*|. (39) 




(t) 


This can be obtained from the following derivation: 
Considering there are mp elements in n I_|_ with 
mp < K, then \I+\J^^\ = K - mo , \ji*\l+\ = 
m* — mo, since m* > K, m* — mo > K — mo- 
(|43|l now becomes: 


2||m 




t,x. 


■+\^+ 


(45) 


< 2|J'|*^\I+| min |wt,i|^46) 






(47) 


‘^\\'^t,jf\x+ ®J<‘)\I+II2 


< 2Wwt — X* 


(48) 


where ( |46l l is from the selection of At in ( [TT] ), i.e., At < 

Plugging the above three items, i.e., ( [T5] l, ( [4T] ) and ( |48] l into 
p4|, we have 


\\9t-x*\\l < {42m*)\\wt - X 

Combing < |49] t and ( [28) t, we have 

{a^-2a){l-S) 


*l|2 

2- 


|mt+i - a;*||2 < 


< (4 +2m*) 


1 


Ot - X* 


(49) 


(50) 


(a2 - 2a)(l - d) 


\wt - a:*||2, (51) 


where we have imposed a G (0, 2). We further impose 
{a^ — 2a)(l — 6) 


(4 +2m*) 


< 1 , 


which is 


ia^-2a)il-S) 


< 


1 


4 + 2m* 


Following this, we have 


|a-l| < t 1- 


(3 + 2m*) 


(1 — 5) (4 -b 2m*) ’ 


(52) 


(53) 


(54) 
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which is 


Following the derivation in Section |III-A| by multiplying a 
factor of 2 , we have 


gmax (3 + 2 to*) 

1 - W 1 - ;-r < a 


(1 — ^) (4 + 2m*) 


\\Wt+l - X 


< 2 


1 + 


(of^ — 2a)(l — 5) 


\et-x* 


In order to show the existence of a, we need 


+2a^|iA'(AA')-^e||^ (63) 

(a^ — 2a)(l — S) 


< (8 +4m*) 


1 


This requires 


ginax (3 + 2 m*) 

“ (1^ (4 + 2m*) ^ 
^(3 + 2 m*) 


0 < 5 < 1 - e" 


(4 + 2m*) ■ 


(56) 


(57) 


x\\et-x*\\l + 2a^A-^V'e\\l ( 64 ) 
(a^ — 2Qf)(l — S) 


< (8 +4m*) 


1 


x\\et-x*\\l + 


2a^ 


• m£LX _ (2771 ~1“4) 

assuming < faFF+ay- 

The above derivation leads the following theorem: 

Theorem 1 Let be a sequence generated by the GAP 

algorithm presented in Section [^/or y = Ax, with x* being 
K-sparse signal satisfying y = Ax*. Assume that the sensing 
matrix A satisfies the RIP 


For convergence, we need 

ia^-2a)il-S) 


(8 +4m*) 


1 


12 - 


< 1 . 


(65) 


( 66 ) 


This is 


/ e“=>'’'(7 + 4m*) 

y (1 —^)(8 + 4m*) 

< 1 - 


^ emax(7_^4m*) 


n ^ X (2to* + 3) ^jjjax 

0 < Om*+K <1 m * I ’ 

(2m* + 4) 


(58) 


where 


(1 — (5)(8 + 4m*) 
In order to prove that there exists a, we need 

gmax(7_^4m*) 

1 - ^^ > 0 , 


• m* > K is the sparsity of {9t}^i generated by GAP; 

• is maximum eigenvalue of AA^ with < 

(27n*+4) 

(2m*+3)’ 

and the step size a 


which leads to 


(1 — i5)(8 + 4m*) 
4m* + 7 


5<l- 


^ ^ + 2 m*) ^ ^ ^ L e^^^{3 + 2m*) 


(1 — (5)(4 + 2m*) 


(1 — (5)(4 + 2m*) ’ 


4m* + 8 

The reconstruction error is bounded by the noise term 

2 

_ ) ■ 

(1 —<5)(8+477i 


( 68 ) 


(69) 


o 7 l , /l e-"-‘>^(7+4m*) y 
2a^ + (l-<5)(8+4m*) J 


I 2 < 


then 


\wt-x*\\l<y\\\wQ-x*\\l, (59) 


where 


71 = (4 +2m*) 


1 + 


(a2-2a)(l-<5) 


< 1 , (60) 


e||2-(70) 

giiiiii " giiiiii 

The above derivation leads to the following theorem in the 
noisy case: 

Theorem 2 Let be a sequence generated by the 

algorithm presented in Section |^/or y = Ax + e, with x* 
being K-sparse signal satisfying y — Ax*. Assume that the 
sensing matrix A satisfies the RIP 


and S = 5m->+K- {tOt} converges to the true signal x*. 

B. Noisy Case 
Recall 

tot+i-a;* = 0t-x*-aA^{AA^)-^Aiet-x*) 

+aA^(AA^)-^e. (61) 

Taking £ 2 - 1101 ™ on both sides and using Lemma 

\\wt+i-x*\\l < 2\\et-x*\\l 

+2(a^ - 2Q!)||A-5UA(6»t - x*)\\l 

+2a2||AT(AAT)-ie||2. (62) 


0 < (5, 


m*-\-K 


< 1- 


4m* + 7 


( 71 ) 


4m* + 8 

where 

• m* > K is the sparsity of {0t}“ generated GAP in 
Section 


(4m*+8) 

(4m*+7)’ 

and the step size a 


is maximum eigenvalue of AA and e” 


< 


^ e™‘"’'(7 +4m*) ^ ^ ^ ^ L e“‘"’‘(7 + 4m*) 


(1 — 6){8 + 4m*) 


\wt - x*\\l < ylllwo - x*\\l + 


(1 — 5)(8 + 4m*) ’ 


2a^ 


( 72 ) 


then 
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where 

72 


(8 + 4to*) 


{a^-2a){l-5) - 

gmax 


<1, (73) 


and e™“ is the minimum eigenvalue of A . {i«t} converges 
to the true signal x* until reaching some error bound. 


It can be observed that both 5 and a have a tighter bound than 
them in the noiseless case as well as 


C. Noise Estimation 
In the noisy case, 

wt+i = 6 »t - aAT(AAT)^iA( 0 t - x*) + aA^{AA^Y^e. 

(74) 

Left-multiplying A on both sides of ( |74| l, 

Ai(Jt_|_i = A9t — aA{0t — X*) + ae. (75) 

Consider limt_).oo dt = x* and the noise can thus be estimated 
via 

a 

IV. Relation to Adaptively Iterative 
Thresholding Algorithms 

When the (AA^)^^ is not used in (j^, GAP will degrade 
to the adaptively iterative thresholding (AIT) algorithm as 
investigated in 0, GT), Gg, Gg. The updating equations 
become 

tut+i = Ot+aA^iv-AOt), (77) 

9t = lUt © maxi 1 - T^,01 . (78) 

I l^’il J 

Note that the RIP condition derived in GD is based on 5^k+i 
while in our paper, we derive the RIP condition for AIT and 
GAP based on 6m* +k- When m* = K, our condition is based 
on 52k^ thus looser than the conditions in [TSl . We extend our 
RIP condition developed for GAP to AIT based on Sm*+K 
below. 


A. Noiseless Case 

Following the derivation in Section |III-A[ we have 

Wt+i -X* = 9t-x* + aA^ {y - A9t) 

= 9t-x* + aA^ {Ax* - A9t) 

= 9t-x* - aA^A{9t - x*). (79) 

Taking ^ 2 -norm on both sides, we have 

\\wt+i-x*\\l = \\9t- x*\\ 1+Y\\A^A{9t-x*)\\l 
-2a\\A{9t - x*)\\l (80) 

Let us introduce the following lemma; 

Lemma 3 The non-zero eigenvalues of AA^ and A^A are 
same. 

Proof: The singular value decomposition of A gives: 

A = PSQT, (81) 


with P G S G and Q G Following 

this: 

AA^ = P(SS^)P^, (82) 

A^A = Q(STS)QT. (83) 

Therefore, the eigenvalues of AA^ and A^A are the same. 


Recall that the maximum eigenvalue of AA^ (thus 

A^A), from Isoj l we have 

< [1 + - 2a(l - J)] ||0* - 

(84) 

Integrating with ( |49] i, we have 

\\wt+i-x*\\l < [l + a2(e“"")"-2a(l-5)] (4 + 2m*) 


x\\wt-x*\\l. (85) 

On the other hand, recall ( |2T] i 

||A^A(et - x*)\\l = (Ot - x*YA^AA^Ai9t - x*) 

= {9t - a;*)^A^UAU^A( 6 >t - x*) ( 86 ) 

= \\A-^V^A{9t-x*)\\l 

< gmax|iuTA(6/, -tc*)||2 (87) 

<e’^^^{l + S)\\9t-x*\\l ( 88 ) 

Combing with ( [49] i, we have 

\\wt+i-x*\\l < [l + a2(l + <5)e“""-2a(l-5)] 

x(4-f 2 m*)||m 4 -a;*||^. (89) 

Along with ([85]l, for convergence, we need either 


[1 + - 2a(l - ,5)](4 + 2m*) < 1, (90) 

or 

[1 + a2e““(l + (5) - 2a(l - <5)](4 + 2m*) < 1. (91) 


Separately consider these two cases: 
a) Equation (|90|) can be simplified to 


(!-<?) 
^gmax ^2 

Following this 

l-<5 


2 ^ (2m* -I- 4){1 - 5)2 - (2m* + 3 )(e™‘‘x )2 


(2m* -I- 4)(em=>-’")'l 


^gmax^2 


1- I 1- 


{2m* + 3 )(e“'‘x )2 
{2m* + 4)(1 — (5)^ 


< a 


< 


1-6 


^gmax ^2 

and we need 


/ ( 2 m* + 3 )(e™^x )2 

“^V “ ^*^+4j(r-5)2 


(2m* + 3)(e-x)2 

(2m*+4)(l-5)2 ^ ’ 


which is 


0 < 5 < 1 - e" 


. V 2 iii* 
Y2m* 


where we assume 

V2m*+3 


(92) 


(93) 


(94) 
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b) Equation ( |9T] ) can be simplified to 
(1-5) 


a — 


< 


e'nax(l + 

{2m* + 4)(1 - 5)2 - (2m* + 3)(1 + 5)e" 

(4 + 2m*)(e“ax)2('i_^^)2 


Following this 

emax(l + S') 

^ ( 1 -^) 
e“ax(i + S) 

and we need 


1- t 1- 


(2m* + 3)(1 + 5)6" 


(2m* + 4)(1 — 5)2 


1 + -i /1 — 


(2m* + 3)(1 + 5)e” 


(2m* + 4)(1 — 5)2 


, ( 2 m*+3)(l + 5)e“a=^ „ 

(2m*+4)(l-5)2 


which is 


C2 = 


2m* + 3 


which is 


gmax ^ ^ 2m* + 4 


0 < 5 < 


of { 61 \ generated by {78). 


If the sensing matrix A satisfies the RIP 

. \j2m* + 3 


0 < 5 




< 1 - e" 


\/2 


m 


with e™ax <; ^2™*+4 ^ ^ 

\/2m*+3 ^ 


a-5) 

^gmax ^2 

( 1 -^) 
^gmax ^2 


1- 


( 2 m* + 3 )(e™ax )2 
(2m* + 4)(1 — 5)2 


< a 


< 


1 + t 1- 


( 2 m* + 3 )(e™ax )2 
(2m* + 4)(1 — 5)2 


or A satisfies the RIP 


0<5-*+^ < 2 ^* ^ 3 , 


and the step size a 


(95) 


< a 


e'nax(l + S) 

(1-5) 


1- Wl- 


< 


'(1 + 5) 


1 + t 1 - 


(2m* + 3)(1 + 5 )e™a 
(2m* + 4)(i — 5)2 

(2m* + 3)(1 + 5)e 


< a 


(2m* + 4)(1 — 5)2 


with e“ax < 2m*+3 » 


\\wt-x*\\l < 7‘J|u;o-a;*|l 2 , 


(106) 


(107) 


where 


(96) 

(97) 


p ^ ^ ^ 2 + Cae'^ax _ ^g2(-gmax)2 8c2e"aax 


(99) 


2m* + 4 
For we need 

2 + 626 "“=' - ^c^(e“ax)2 8c2e“ax 
0 < —^-^+^1- - -^-• < 1, (100) 


( 101 ) 

62 2m* + 3 

In this case, we only consider e™ax > (1 + 5) (otherwise, 
please refer to case a)), together with ( | 101 | l, 

1 


( 102 ) 


2m* + 3 

The above derivation leads to the following theorem for 
AIT in the noiseless case: 

Theorem 3 Let {wt}^i be a sequence generated by 
the AIT presented in (|77|)-([75|) for y — Ax, with x* 
being K-sparse signal satisfying y = Ax*. Let e™ax 
maximum eigenvalue of AA^. m* > K is the sparsity 


7no = 73 = [1 + a^(6“'^’")^ - 2a(l - 5)] (4+2m*) < 1, 

(108) 

if ( |7Q3p -( |70?| are satisfied, and 

7 no = 74 = [1 + + 5) — 2q(1 — 5)] (4+2m*) < 1, 

(109) 

if { 105\ - ^f0^ are satisfied with 5 = 5m*+jf- {mt} 
converges to the true signal x*. 

Note that the bounds of RIP-5 in ( |105P is looser than those 
in ( fTO^ . 

B. Noisy Case 
In the noisy case, 

— X* = 9t — X* — aA^A{9t — x*) + aA^e. (110) 
Taking 72 -norm on both sides and using Lemma 
\\wt+i - a :*||2 < 2\\9t - a :*||2 + 2 a 2 ||ATA(e* - at *)||2 

-4a||A(0t-a:*)||2 + 2 a 2 |jATe|| 2 . (Ill) 
For the noise term, using ([ 2 T]i 


2a2||A' e||^ = 2a2e ' AA ' e = 2a2e ' UAU ' e (112) 


= 2a2||A5U^e||2 

2 ^max I IITtT^||2 


< 2 a 6 


U' e||2 = 2a^e 


2^max|| 


(113) 

(114) 


(103) 


Without listing all the details, we provide the convergence of 
AIT under the noisy case. 

Theorem 4 Let be a sequence generated by the AIT 

presented in ([77|)-([75|) for y — Ax + e, with x* being K- 
sparse signal satisfying y = Ax*. Let e™ax maximum 
eigenvalue of AA^. m* > K is the sparsity of {9t} generated 

If the sensing matrix A satisfies the RIP 

. v/4m* + 7 

V4m* + 8 ’ 


0 < 5m-+K < 1 — 6 ” 


(115) 


with e^ax ^ +4m*-|-8 ^ 

v4m*+7 ^ 


(104) 


(105) 


(l-<^) 

^gmax ^2 

. (i-<^) 

^gmax^2 


1 - Wl- 


(4m* + 7 )( 6 ”aax )2 
(4m* + 8)(1 — 5)2 


< a 


1 + A /l — 


(4m* + 7 )( 6 ™ax )2 
(4m* + 8)(1 — 5)2 


(116) 
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or the sensing matrix A satisfies the RIP 

1 


0 ^ ^m*+K ^ 


Am* + 7 ’ 


and the step size a 

( 1 -^) 
emax(l + S) 

a-s) 


1- Wi- 


(4to* + 7)(1 + (5)e" 


(Am* + 8)(1 — (5)^ 


< a 


< 


^(l + <5) 


1 + Wl- 


(4m* + 7)(1 + 5 )e“a 
(4m* + 8)(1 — 5)^ 


with < 5 ^^, then 


\wt - x*\\l < 7‘J|u7o - 35*112 + 2a^e" 


where 


0 < 5-iK+i < 


1 


(117) 


(118) 


(119) 


7 ns = 75 = [1 + - 2 a(l - <5)] (8 + 4m*) < 1 , 

( 120 ) 

if ( |775| >-( |776| are satisfied, and 

7ns = 76 = [l + + 5) — 2a(l — 5)] (8 + 4m*) < 1, 

( 121 ) 

(/'( |777| l-( |77gp are satisfied, with S = 6m*+K- converges 
to the true signal x* until reaching some error bound. 

C. Relation to the Results in m 

Wang et.al G!) derived the RIP condition of AIT based on 
5^k+i and the condition is 


. In the coded aperture compressive hypespectral imag- 
ing p7) , pO) and video compressive imaging systems 0 , 

1311, AA ' is a diagonal matrix and thus very easy to compute 
the inverse. 

V. Convergence Rate Comparison Between GAP 
AND AIT 

We now compare the convergence rates of GAP and AIT. 
In all the above theorems, each one has a convergence rate 
Since 0 < 7 ^ < 1, a smaller 7 ^ will lead to faster 
convergence. In the following, we compare 7 ^ at different 
cases. For each case, there is one smallest 7 * that leads to 
fastest convergence, i.e., 7 * is chosen to be the minimum of 
the 7 i in the error bound. 

• For GAP, a = 1 leads to the smallest { 7 i}f=i 

[ginax _ (1 _ J)] 


7 * “=^ (2m*+4)1 


‘S' ( 4 „- + 8)!571++i)!, 

' ^ '' ' ^ma.v 


(123) 

(124) 


For AIT, we have different values below: 

[(emax)2 _ (1 _ 5)2] 


(1-5) 

Ci— -- '—r^ 

/•gmax\2 ^ . 

73 * = (2m*+4)i 


_ (1-5) 

74 = (2m* + 4) 


^gniax ^2 
]gmax(i 5 ) _ _ 5 ) 2 ] 

emax(l -p 5) 


(125) 


(126) 

^ ^ ^ ^ [(e“ax)2 - (1 - 5)2] 

75 * = (4m*+8) -(^ - 11, (127) 


( 122 ) 


V 2 A + 4' 

Our results, by contrast, are based on Sm-^+K, with conditions 
in ( |103| l or ( |105| ). When m* = K, our results are based on 
S2K- 

Our results in ( |108[ ) and ( |109| l imply that the step-size a 
that leads to the fastest convergence of AIT is not always at 


76 


. ( 1 -^) 

- emax(i + 5) 


(4m* + 8 ) 


^gmax ^2 
]gmax (i 5) _ _ 5)2] 

emax(l -p 5) 


(128) 


a = 1, which is obtained in |18|. In our case, the optimal 
step-sizes that lead to fastest convergence of AIT depend on 
the RIP condition (and/or the maximum eigenvalue) of the 
sensing matrix. On the other hand, our convergence results 
require conditions on the eigenvalues of sensing matrix as well 
as their relationship with RIP, while the results in p 8 ) only 
require the RIP. In addition, the convergence rate investigated 
in our paper is for {ru*}, while the results in are derived 
for {Ot}. 

Regarding the computational cost, if A^(AA^)“^ is pre¬ 
computed and savec^l the computational workload for GAP is 
the same as AIT, i.e., 0{MN) p^ . For the comparison of 
AIT with other algorithms, please refer to m- On the other 
hand, in a lot of real compressive imaging systems, AA^ is 
a diagonal matrix |21|, 1^, |31|. For example, AA^ = I 


We now compare GAP and AIT under the fastest con¬ 
vergence. Since 6 G (0,1), and it is eas}0 to show that 

e^iax > _ 5)^ 

(1 - Sf < (1 - 5)e”“ 

„max'\2 „max/ 


[(gmax )2 _ gmax(j _ 5 )] ^ ](eniax )2 _ _ 5 ) 2 ] 


^gmax^2 ^gmax^2 

(l-5)^<(l-5)(l + 5) 

[(gmax)(i + 5)-(i + 5)(i-5)] 

(gmax)(l 5 ) 

[gmax(i_^5)_(l_5)2] 


(129) 


< 


^{1 + S) 


Along with { 7 *}f _4 above, we have 


and 


if the permuted Hadamard matrix is used in the single-pixel 
camera and the lensless compressive camera [ [^ , p 8 ) , 

* Since AA^ is usually a sparse matrix, there are spai'se LU or Cholesky 
factorization for AA^, and therefore, if the factorization is pre-computed 
and stored, A^(AA^)“^'i; can be easily computed in each iteration, with 
V denoting a vector. 


7i < 74: 72 < 76) 


7i < 73 . 72 <75- 


(130) 


(131) 


(132) 


Therefore, the convergence rate of GAP is faster than that of 
AIT. 

^This can be proved as follows, (1 — <5)||a;||| < ||A3:||| = a;^A^Aa: = 
(cc, A^^Aa:} < e“^x||^||2^ 













































VI. Simulation Results 


q= 0.9, M=300, N=512, K=20, m*=20 


a=0.9, SNR = 60dB 


A set of simulation experiments are conducted in this 
section to demonstrate the validity of the theoretical results. 
Moreover, we compare the convergence of GAP and AIT 
under the same condition. 


A. Experimental Setup 

In the following experiments, we set M = 300, N = 512. 
AT = 20 is used in the true sparse signal a;*. The selection of 
m* is usually set to m* = K. The nonzero elements of x* 
are generated randomly from the standard normal distribution. 
The sensing matrix A is generated from i.i.d. norm distribution 
A/^(0,1/M) |T^ . Other types sensing matrix are also used and 
similar observations have been obtained. 

When the noise is consider, (signal-to-noise ratio) SNR = 
60dB is used. Different step-sizes a = {0.9,1.0,1.1} are 
conducted and the results are shown in corresponding figures. 


0=0.9, M=300, N=512, K=20, m*=20 






Iteration, t 



Iteration, t 



Fig. 1. Linear convergence of GAP and AIT in both noiseless (left) and 
noisy case (right with SNR = 60dB). Step-size a = {0.9,1.0,1.1} for each 
row. 







Fig. 2. Linear convergence of GAP and AIT in both noiseless (left) and 
noisy case (right with SNR = 60dB) with binary sensing matrix. Step-size 
a = {0.9,1.0,1.1} for each row. 


until reaching some error bounds. We can observed that in 
both cases, GAP converges faster than AIT for every step- 
size. In the noisy case, the error bounds that the recovered 
signal achieved of GAP are smaller than those of AIT. These 
results verify the theoretical analysis in our theorems. Similar 
observation can be found when the sensing matrix is binary 
distributed (Figure]^ | [T6| . 


C. Different Selections of m* 

In real cases, usually, we have no prior knowledge of the 
sparsity of the true signal, i.e., K is unknown. Figure plots 
the reconstruction errors (stopped at ||u;( — x *\\2 < 10“® in 
the noiseless case and at ||u;t — tc*||| < 10“^ in the noisy 
case) with different selection of m* when K = 20. It can 
be seen that when m* € [20,62], both AIT and GAP can 
provide good estimates of the signal. However, GAP can still 
reconstruct the signal even when K > 62, while AIT diverges 
at these sections of m*. 


B. Convergence Rate Justification 

Figure demonstrates the linear convergence of GAP and 
AIT. In the noiseless case (the left three plots in Figure [^1, 
Wt converges to the original sparse signal with high precision, 
e.g., \\wt — x *\\2 < 10“® with about 40 iterations for GAP 
and about 60 iterations for AIT. In the noisy case (the right 
three plots in Figure [^, jjtut — a :*||2 decays exponentially 


D. Robustness of the Signal Sparsity 

We conduct the robustness of the algorithm regarding to 
the signal sparsity. The same number of measurements M = 
300 are used in the experiments with different K, the sparsity 
of the true signal. Figure plots the reconstruction errors of 
GAP and AIT at different K. It can be found that AIT can 
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a=1, M=300, N=512, K=20 








Fig. 3. Different selections of m* for GAP and AIT in noiseless (top) and pjg 5 Linear convergence of GAP and AIT in both noiseless (left) and noisy 
noisy (bottom, with SNR = 60dB) cases. Step-size a = 1 is used. cases (right with SNR = 60dB) with K = 15. Step-size a = {0.9,1.0,1.1} 

for each row. 




Fig. 4. Reconstruction errors with diverse sparsity (K) of the signal x* in 
noiseless (top) and noisy (bottom, with SNR = 60dB) cases. Step-size a = 1 
is selected. 


provide good estimates when K < 20, while GAP provides 
good reconstructions up to AT = 42. This is further verified in 
Figures 1^ and 1^ In Figure]^ AT = 15 is used to generate the 
signal and both GAP and AIT work well. By contrast, when 
AT = 35 is used to generate the signal (Figure |^, GAP can 
reconstruct the signal while AIT fails. Similar observations 
can be found with other numbers of measurements. 

E. Noise Estimation 

As stated in Section [IlTC| GAP can estimate the noise via 
the sequences of {wt} and {6t}. In our simulation, we add 
noise with different standard deviations (std.) and run GAP 
until it converges to an error bound. The noise is estimated 
based on We compute the standard deviation of this 

estimated sequence of noise and compare with the truth in 
Figure It can be observed that GAP can estimate the noise 
accurately in a large range. 

E Image Compressive Sensing 

Now we test the performance of GAP and AIT on the 
image compressive sensing problem. Consider the “Lenna” 
image with size 256 x 256 and we used 6554 measurements 
(10% of the image pixels). Reconstruction is based on the 
sparsity of DCT (Discrete Cosine Transformation) coefficients 
of the local overlapping patches p^ , which has been shown 
to perform better than the global wavelet transformations ED- 
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a=1,M=300, N=512, K=35, m*=35 





Fig. 6. Linear convergence of GAP and AIT in both noiseless (left) and noisy 
cases (right with SNR = 60dB) with K = 35. Step-size o = {0.9,1.0,1.1} 
for each row. Note AIT fails to reconstruct the signal but GAP still estimate 
the single accurately. 




Fig. 8. PSNR of reconstructed images and the reconstruction error per 
iteration in the noiseless case. Lenna image is used with size 256 x 256 
and 10% (of the image pixels) measurements are used. 




Fig. 9. PSNR of reconstructed images and the reconstruction error per 
iteration in the noisy case with SNR = 60dB. 



Fig. 7. Noise estimation via GAP compared with the truth. 


The Gaussian sensing matrix is used and both noise-free 
and noisy cases (SNR = 60dB) are considered. The PSNR 
(peak signal-to-noise ratio) of reconstructed images and the 
reconstruction errors are plotted versus iteration in Figure 
(noiseless case) and Figure (noisy case). The reconstructed 


images obtained by GAP and AIT are shown in Figure 10 


and Figure [TT] respectively for the noiseless and noisy cases. 
It can be seen that in both cases, GAP again performs 
better than AIT. Similar observation can also be found in the 
video compressive sensing ED and hyperspectral compressive 



Fig. 10. Reconstructed images in the noiseless case. Lenna image is used 
with size 256 x 256 and 10% (of the image pixels) measurements are used. 


AIT GAP 



Fig. 11. Reconstructed images in the noisy case with SNR = 60dB. 
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sensing 113 problems. 

VII. Conclusion 

Investigated here is an adaptively generalized alternating 
projection algorithm with applications to compressive sensing. 
The linear convergence of the algorithm has been derived 
based on the restricted isometry property condition of the 
sensing matrix. The theoretical analysis has also been extended 
to adaptively iterative thresholding algorithms. Both theoret¬ 
ical analysis and experimental results demonstrate that the 
generalized alternating projection algorithm converges faster 
than the adaptively iterative thresholding algorithm. 

In our experiments, we have found that sometimes, a larger 
step-size of the generalized alternating projection algorithm 
will lead to faster convergence, even when a approaches 
2. The conditions derived in this paper are sufficient condi¬ 
tions for the generalized alternating projection algorithm to 
converge. It may be possible that the generalized alternating 
projection algorithm converges within a larger range of step- 
sizes than that derived in this paper. 
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